I had an offline discussion with a stakeholder who also raised a valid question that can be added to the document:

Q: "How is Concourse on K8s determining the state of the pods?"
A: There are liveness and readiness probes defined in the chart, which make a http request to the /api/v1/info endpoint.
The idea of the change would be to have a more dedicated endpoint that could build a bit on that static endpoint checking by also considering the status which can change dynamically.

Thanks for the question I will also add this to the document when a bit more questions are gathered!

Also part of an offline follow-up with another user:

"You can think of Google status health" to have a more red/green status pointing towards potential problems with the application.

I think a GUI change is a bit out of scope of this RFC, albeit this RFC would enable this to be easily extended in the UI, so it is worth writing it done as a possible future follow-up

Totally for this. A lot of my questions come around implementation, which I see already written down in the POC PR concourse/concourse#4818. I think it would be nice for this RFC to define specifically what we want the Health JSON response to look like.

A Concourse web node is made up of a bunch of micro-service-ish components. We could potentially display the health of all of these components (see components.go). There may be some exceptions in that file, but most of these components are run "globally" across one of the web nodes based on workload the web node is handling. They're load-balanced!

There are some services on the web node that are not load-balanced, like the TSA and API. Those are always running on all web nodes.

A detailed health response could look something like this, which I think would accurately describe the entire Concourse cluster:

{
    "status": "...",
    "workers": {
        "worker-1": {
            "baggageclaim": "...",
            "garden": "..."
        }
        ...
    },
    "web-nodes": {
        "web-1": {
            "api": "...",
            "tsa": "...",
            "db-connection": "...",
            ...
        }
        ...
    },
    "global-components": {
        "log-collector": "...",
        "lidar": "...",
        "secret-management": "...",
        "scheduler": "..."
        ...
    }
}

I wouldn't expect an initial PR to fully implement all of that though. I think this RFC could clearly define what we want the end goal to look like and then slowly work towards it through multiple PR's. WDYT?

I agree, I really like the idea and I am all for having an easy to reach status board for all of the components. One of the key questions that come to mind is when the overall status should change to not healthy, as albeit each component having its share of work to be done, if they flap, or are unstable it shouldn’t mean the instance is not operational, but rather somewhat degraded. So more, or less building on what you wrote, it would be great before closing the RFC to have the json response and the conditions that are hard requirement for a healthy instance figured out. Thanks to all for the feedback, I like the overall direction of the discussions here. Once we have q couple of more comments, I will add all the discussions to the document.

If you plan to reuse the same endpoint for Kubernetes health checks, you can introduce a parameter to differentiate between web and worker nodes. For example:

• /health?component=web
• /health?component=workers

It could also be extended to the pod level, such as:

• /health?component=workers-1
• /health?component=workers-n

This way, Kubernetes can identify and restart individual pods if they become unhealthy.

The committers listed above are authorized under a signed CLA.

• ✅ login: Kump3r / name: Kump3r (3b2ae43, 9320105)